Orthogonality
Lecture 16
Recap
$$ % Colors
% Coordinate vectors and matrices
% Common sets
% Abstract vector symbols
% Norms / absolute value
% Optional: dot product spacing (looks nicer in slides)
% Operators $$
Dot Product
- The dot product (also called the scalar product or inner product) is defined for two vectors \(\vec{u},\vec{v}\in\mathbb{R}^n\).
- The result is a scalar (a real number).
- If \[ \vec{u}=\langle u_1,\ldots,u_n \rangle, \quad \vec{v}=\langle v_1,\ldots,v_n \rangle, \] then \[ \vec{u}\!\cdot\!\vec{v} = u_1v_1+\cdots+u_nv_n. \]
- The dot product measures length and angle.
Properties of the Dot Product
- \(\vec{u}\!\cdot\!\vec{v}=\vec{v}\!\cdot\!\vec{u}\) (commutativity)
- \(\vec{u}\!\cdot\!(\vec{v}+\vec{w})=\vec{u}\!\cdot\!\vec{v}+\vec{u}\!\cdot\!\vec{w}\) (distributivity)
- \((c\vec{u})\!\cdot\!\vec{v}=c(\vec{u}\!\cdot\!\vec{v})\) (compatibility with scalars)
- \(\vec{u}\!\cdot\!\vec{u}\ge 0\), and \(\vec{u}\!\cdot\!\vec{u}=0\) iff \(\vec{u}=\vec{0}\) (positivity)
- \(\left\lVert \vec{u} \right\rVert^2=\vec{u}\!\cdot\!\vec{u}\) (length from dot product)
- \(\displaystyle \cos\theta=\frac{\vec{u}\!\cdot\!\vec{v}}{\left\lVert \vec{u} \right\rVert\,\left\lVert \vec{v} \right\rVert}\) (angle from dot product)
Orthogonal Projection
- Let \(\vec{u},\vec{v}\in\mathbb{R}^n\) with \(\vec{v}\neq\vec{0}\).
- Decompose \(\vec{u}\) relative to \(\vec{v}\): \[ \vec{u}=c_1\vec{e}+c_2\vec{n}, \] where \(\vec{e}=\frac{\vec{v}}{\left\lVert \vec{v} \right\rVert}\) is the unit vector in the direction of \(\vec{v}\) and \(\vec{n}\perp\vec{v}\) in the plane \(W=\mathop{\mathrm{span}}\{\vec{u},\vec{v}\}\).
- The scalar \(c_1\) measures the component of \(\vec{u}\) along \(\vec{v}\).
- The vector \(c_1\vec{e}\) is the projection of \(\vec{u}\) onto \(\vec{v}\), denoted by \(\operatorname{proj}_{\vec{v}}\vec{u}\).
Orthogonal Projection
Orthogonal Projection Formula
- Start with \(\vec{u}=c_1\vec{e}+c_2\vec{n}\).
- Take the dot product with \(\vec{v}\): \[ \vec{u}\!\cdot\!\vec{v} = c_1(\vec{e}\!\cdot\!\vec{v}) = c_1\left\lVert \vec{v} \right\rVert. \]
- Hence \[ c_1=\frac{\vec{u}\!\cdot\!\vec{v}}{\left\lVert \vec{v} \right\rVert}. \]
- Therefore, \[ \operatorname{proj}_{\vec{v}}\vec{u} = c_1\vec{e} = \frac{\vec{u}\!\cdot\!\vec{v}}{\left\lVert \vec{v} \right\rVert^2}\,\vec{v} = \frac{\vec{u}\!\cdot\!\vec{v}}{\vec{v}\!\cdot\!\vec{v}}\,\vec{v}. \]
Example
- Let \(\vec{u}=(1,0,-1)\) and \(\vec{v}=(-2,2,1)\).
- Compute \(\vec{u}\!\cdot\!\vec{v}\) and \(\left\lVert \vec{v} \right\rVert\). Answer two numbers on iClicker separated by a comma.
- Is the angle between \(\vec{u}\) and \(\vec{v}\) parallel, opposite, acute, obtuse, or perpendicular?
Scan the QR code or go to join.iclicker.com/MBNJ.
Example (continued)
- Let \(\vec{e}=\frac{\vec{v}}{\left\lVert \vec{v} \right\rVert}\) be the unit vector in the direction of \(\vec{v}\), and let \(\vec{n}\) be a unit vector perpendicular to \(\vec{v}\) in \(W=\mathop{\mathrm{span}}\{\vec{u},\vec{v}\}\).
- Decompose \[ \vec{u}=c_1\vec{e}+c_2\vec{n}. \]
- Take the dot product with \(\vec{v}\) and simplify the right-hand side.
- Use your previous results to compute \(c_1\). Enter your answer on iClicker.
Scan the QR code or go to join.iclicker.com/MBNJ.
Inner Product Space
Warm-up
- How do we define a “dot product” between \[ -1+x+x^2 \quad \text{and} \quad 3+3x \] in the space of polynomials?
- Interpret them as coefficient vectors: \[ (-1,1,1) \quad \text{and} \quad (3,3,0). \]
- Then \[ \langle -1,1,1 \rangle\!\cdot\!\langle 3,3,0 \rangle=0. \]
- Even without a geometric angle, this value still has meaning.
Motivation for Inner Product
- In abstract vector spaces, we can still define a dot product (typically called an inner product).
- However, we may no longer interpret it geometrically (no physical angle).
- Instead, we interpret \(\vec{u}\!\cdot\!\vec{v}\) as measuring similarity or correlation between vectors.
- In the polynomial example, the dot product is \(0\), meaning the vectors are uncorrelated (orthogonal).
Inner Product
- Just as vector spaces are defined axiomatically, we define an inner product axiomatically.
- For vectors \(\vec{u},\vec{v}\) in a vector space \(V\), an inner product is a function \(\left\langle \vec{u},\vec{v} \right\rangle : V\times V \to \mathbb{R}\) satisfying the following properties:
- Linearity (in the first variable):
\(\left\langle a\vec{u}+b\vec{w},\vec{v} \right\rangle = a\left\langle \vec{u},\vec{v} \right\rangle + b\left\langle \vec{w},\vec{v} \right\rangle\) for all scalars \(a,b\). - Symmetry:
\(\left\langle \vec{u},\vec{v} \right\rangle = \left\langle \vec{v},\vec{u} \right\rangle\). - Positivity:
\(\left\langle \vec{u},\vec{u} \right\rangle \ge 0\) for all \(\vec{u}\in V\). - Definiteness:
\(\left\langle \vec{u},\vec{u} \right\rangle = 0\) if and only if \(\vec{u}=\vec{0}\).
Inner Product for Functions
- What is a natural inner product between functions?
- Think of a function \(f(x)\) as an infinite-dimensional vector, where each “coordinate” is indexed by \(x\).
- For functions \(f(x)\) and \(g(x)\), we might try an infinite sum \(\sum_{x\in\mathbb{R}} f(x)g(x)\), but uncountable sums are not defined.
- Instead, we replace the sum with an integral and define \[ \left\langle f,g \right\rangle = \int_{\mathbb{R}} f(x)g(x)\,dx. \]
- It is straightforward to check that the above definition satisfies all the required axioms for inner products.
Inner Product Space
- A vector space equipped with an inner product is called an inner product space.
- In such spaces, \(\left\langle \vec{u},\vec{v} \right\rangle\) measures how two vectors are correlated.
- If \(\left\langle \vec{u},\vec{v} \right\rangle=0\), we say \(\vec{u}\) and \(\vec{v}\) are orthogonal.
- In this course, we mainly use the standard dot product on \(\mathbb{R}^n\) (and later the inner product of functions for Fourier decomposition).
Gram–Schmidt Process
Motivation
- We want a “better” basis for a vector space.
- A basis is orthogonal if every pair of vectors has dot product \(0\).
- A basis is orthonormal if it is orthogonal and each vector has length \(1\).
- If a basis is orthogonal, we can obtain an orthonormal basis by normalizing each vector.
- Given a linearly independent set, how can we systematically construct an orthogonal basis?
- The Gram–Schmidt process gives an explicit algorithm to do this.
Simple Case
- Let \(S=\{\vec{u},\vec{v}\}\) be linearly independent in \(\mathbb{R}^2\) and let \(W=\mathop{\mathrm{span}}\{\vec{u},\vec{v}\}\).
- If \(S\) is not orthogonal, can we replace it with an orthogonal basis of \(W\)?
- Idea: subtract the component that is parallel using orthogonal projection.
Illustration
Gram–Schmidt (Two Vectors)
- Start with linearly independent \(S=\{\vec{v}_1,\vec{v}_2\}\).
- Set \(\vec{w}_1=\vec{v}_1\).
- Remove the component of \(\vec{v}_2\) parallel to \(\vec{w}_1\): \[ \vec{w}_2=\vec{v}_2-\operatorname{proj}_{\vec{w}_1}\vec{v}_2. \]
- Then \(\{\vec{w}_1,\vec{w}_2\}\) is an orthogonal basis of \(W\).
- Normalize if desired to obtain an orthonormal basis.
General Gram–Schmidt Process
- Let \(S=\{\vec{v}_1,\dots,\vec{v}_k\}\) be linearly independent and let \(W=\mathop{\mathrm{span}}(S)\).
- Set \(\vec{w}_1=\vec{v}_1\).
- For \(j\ge2\), remove all components parallel to the previous vectors: \[ \vec{w}_j = \vec{v}_j - \sum_{i=1}^{j-1} \operatorname{proj}_{\vec{w}_i}\vec{v}_j. \]
- Then \(\{\vec{w}_1,\dots,\vec{w}_k\}\) is an orthogonal basis of \(W\).
- Normalize each \(\vec{w}_j\) if an orthonormal basis is needed.